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Abstract. In 2008 the Spanish Government announced the inclusion of an oral section 
in the foreign language exam of the National University Entrance Examination 
during the year 2012 (Royal Decree 1 892/2008, of 14 November 2008, Ministerio de 
Educacion, Gobierno de Espafia, 2008). Still awaiting the implementation of these 
changes, and in an attempt to offer the Government a cost-effective technological 
solution, the CAMILLE Research Group at Universidad Politecnica de Valencia set 
about creating an online language examination platform complying with the official 
examination parameters based on the InGenio Learning Management System. Once 
the system had been created, the researchers implemented several test exams and 
conducted a pilot study with 183 high school students. The quantitative analysis 
of the study shed light on the learners’ attitudes toward using such a system in the 
context of a high stakes examination. 

Keywords: oral assessment, Spanish university entrance examination, InGenio 
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1. Introduction 

Over the last few years, the use of technology in education has brought about 

changes in teaching and learning materials and methodologies. Assessment too has 
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been affected by these changes, and among the official exams that are currently 
working on offering a computerised version is the Spanish University Entrance 
Examination (in Spanish, Prueba de Acceso a la Universidad - PAU). It is a high 
stakes exam that has a considerable impact on the academic and professional future 
of Spanish students. Consequently, there is also a washback effect that leads students 
and teachers alike to focus only on what is assessed by the exam (Saif, 2006), that 
is, grammar, vocabulary, reading comprehension and writing, leaving out listening 
comprehension and speaking. Therefore, the inclusion of oral assessment in the 
PAU foreign language exam would be beneficial for pre-university students as 
more attention would be paid to oral skills which are currently being neglected 
(e.g. listening comprehension and speaking). The inclusion of these two skills was 
announced by the Spanish government by means of a Royal Decree (Ministerio de 
Educacion, Gobierno de Espaiia, 2008). 

As a response to the changes to be introduced in the new PAU and so as to facilitate 
the simultaneous examination of thousands of Spanish students in a cost-effective 
way, the CAMILLE Research Group at Universidad Politecnica de Valencia 
launched a government- funded project aimed at examining the feasibility of a 
computerised PAU exam. To achieve this goal, an online language examination 
platform complying with the official examination parameters and based on the 
InGenio learning management system was developed. The system is known as 
PAULEX. Once the system had been created the researchers implemented several 
test exams and conducted a pilot study with 183 high school students to test the 
affordances of the platform in terms of the creation, management, delivery and 
correction of the PAU foreign language exam. 

Some of the changes that the government would like to implement in the oral 
section of the exam are an opportunity to develop online tools that can optimise the 
performance of such an exam. A similar thing has already happened to a greater 
or lesser extent with other high stakes language exams, such as the Test of English 
as a Foreign Language (TOEFL) and the Cambridge First Certificate in English 
Examination (FCE). This opportunity could materialise, for example, in the form 
of virtual interviews to assess oral skills as an alternative to face-to-face interviews 
so as to grant anonymity. Also, a significant number of computer-based tools can 
be used to optimise other aspects of the PAU exam, from designing the actual 
exam to delivering and correcting them. Technology today makes it possible to 
achieve this without threatening security. Two of the most prominent examples 
that demonstrate the validity and feasibility of a large-scale computerised or online 
test, at least as far as official language tests are concerned, are the aforementioned 
TOEFL and FCE computerised/online tests. 
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Among the advantages of computerising the PAU exam is the protection of both 
the annual exams to be taken and the completed ones, as well as the possibility 
of programming automatic backups. In addition, an electronic format would 
make it easier for raters and subsequent exam reviewers to immediately, safely 
and efficiently access student exams, in addition to a record of the changes 
and modifications made by each user. With regard to the speaking section, 
which is to become part of the final construct of the PAU foreign language 
exam once the announced changes are implemented (Ministerio de Educacion, 
Gobierno de Espana, 2010), the use of an online version would facilitate the 
simultaneous examination of the approximately 19,000 students who take the 
PAU exam annually in the Valencian Community in a more efficient and cost- 
effective way. 

2. Method 

2.1. Context 

Participants consisted of a total of 183 Secondary Education final year students 
(2nd year of Bachillerato) - 52.46% male and 47.54% female - from 3 public and 
3 private high schools in the Valencian Autonomous Region in Spain. Their ages 
ranged between 16 and 19 and they all intended to take the PAU exam. Of these, 
97.3% (178) were Spanish nationals, whereas 2.7% (5) were not. Except for 2 of 
them (1.1%), the rest owned a personal computer and 100% of them reported using 
the Internet on a daily basis from 1 to 2 hours. 

2.2. Procedure 

Participants filled in two questionnaires. The pre-treatment questionnaire was 
aimed at compiling qualitative data and it included demographic information: age, 
gender, experience in the use of Information and Communications Technology 
(ICT), etc. Using the PAULEX platform, students sat an exam in the Virtual 
Learning Environment (VLE) which was an adaptation of the English paper- 
based exam of the 2007 official PAU exam in the Valencian Region. The items 
of this exam were adapted to the online environment and new items were added 
for the assessment of listening and speaking skills. Once students had submitted 
all their answers, they filled in a post-treatment questionnaire which comprised 
16 close-ended questions, based on a 4-point Likert scale, relating to the usability 
of PAULEX and the participants' overall level of satisfaction, and 6 open-ended 
questions relating to the usefulness of such an online exam and their expectations 
regarding the computerisation of the PAU English language exam. 
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The exam tasks included watching several video clips and answering multiple- 
choice listening comprehension questions and “validating” answers before sending 
final responses to the server for correction. As for the speaking section, this was 
assessed by means of an activity based on an appealing image concerning a current 
issue. Students were asked to observe the image for a couple of minutes and then 
answer a question relating to the topic depicted in the image, giving their opinion 
and ideas about it. Once they had thought out what they wanted to say, participants 
recorded a short speech (no longer than 4 minutes) with the platform’s built-in 
media player. 

3. Discussion 

During the test the vast majority of participants showed an exemplary attitude 
and expressed their satisfaction toward the new exam format, a fact that came 
as no surprise bearing in mind that they belong to a generation of digital natives 
(Prensky, 2001) who are accustomed to working and interacting with electronic 
resources. On the other hand, several interesting aspects were observed during the 
experiment. The students were given 60 minutes to complete the simulation instead 
of the 90 minutes officially allocated. This reduction was necessary so as to avoid 
interfering with their daily school duties and to limit the experiment to the time 
allotted to their regular English classes. 


In spite of the time reduction, all the students were able to complete the test within 
the allotted time, completing all the activities which assessed the different skills, 
including listening comprehension and speaking. Students browsed through the 
different parts of the exam, wrote or chose their answers and then validated them, 
section by section. The students’ answers were then stored on a server to avoid loss 
of information in case of system failure or errors during the course of the trial. All 
in all, the online format was very well received by students, who made numerous 
positive comments regarding its practicality. 


Table 1. Statistical data describing overall satisfaction 


I think it is a good programme. 

It is easy to use. 

The programme is fin to use. 

The programme works just as I expect it to, 

I would recommend a friend to use it. 

The programme fulfils my expectations regarding what an 

online test should be like. 

N valid responses 


N 

Min. 

Max. 

Mean 

Standard 

dev. 

183 

1 

4 

3.15 

.553 

183 

1 

4 

3.10 

.579 

183 

1 

4 

2.77 

.712 

183 

1 

4 

2.93 

.700 

183 

1 

4 

2.97 

.670 

183 

1 

4 

3.21 

.612 

183 
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Table 1 summarises the overall satisfaction as reported by the students who 
participated in the survey. As we can see, all of the items enquired about resulted 
in a favourable attitude. 

4. Conclusions 

The changes in the new PAU foreign language exam require solutions for the 
implementation of a cost-effective and feasible way to assess oral skills. The results 
of the survey and the solutions proposed can be considered a valid alternative for 
foreign language assessment in the Spanish University Entrance Examination, 
including the evaluation of oral skills. Moreover, these results pinpointed the fact 
that it is indeed feasible to implement the assessment of oral skills in the PAU 
foreign language exam. Moreover, this computerised version can be easily updated 
and adapted to the specific characteristics and requirements of the new PAU exam 
in terms of content, usability and security, among others. 

As for the students’ opinions, they were optimistic and willing to switch to the 
online format. All of this suggests that the PAULEX platform meets a number 
of conditions that make it suitable for the preparation and implementation of the 
new PAU exam and, specifically, its foreign language section. However, for the 
successful large-scale implementation of this platform, more research is needed 
and further studies should be conducted with a larger sample which might lead to 
more conclusive results: the empirical study described in this article was conducted 
with only 183 students as opposed to the total number of 19,053 students who sat 
the PAU exam in 2015 in the Valencian Community, according to the data provided 
by the Valencian Regional Government (2015). 
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